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(54) Colour correction with consideration of the scene illuminant 



(57) A color image editing system and method uses 
sensor responses in an image capture device to esti- 
mate and separate illuminant and surface reflectance In 
a scene. The system uses the estimated surface reflect- 
ances, along with a desired illuminant SPD, to calculate 
the corresponding XYZ values. To render the images on 
an imaging device , the XYZ values are converted into 
device-dependent display or printer values. The image 
data may be manipulated in a manner that is intuitive to 
humans, such as correcting the image data according 
to changes In lighting and surface color. 

An Image capture device, such as a scanner or dig- 
ital camera, represents a scene as image data that is 
resolved as device-dependent channels, such as RGB. 
For each channel, the spectral reflectances of the sur- 
faces within the scene are described reflectance func- 
tions that are independnt of the lighting function that de- 
scribes the illuminant spectral power distribution (SPD). 
Display operators can independently adjust the surface 
and lighting functions. 
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Description 

The invention is directed toward image processing 
and more specifically towards correcting illumination in 
Images captured by digital media, sucli as scanners or 
digital cameras. 

Commercially available color image editing (CIE) 
systems are part of large graphics systems, such as 
desktop publishing systems that allow users to generate 
digital representations of color images, edit digital rep- 
resentations for visual appeal, and reproduce digital 
representations on different media. These systems in- 
clude image capture devices, such as digital cameras 
and scanners, image processing hardware, and color 
displays and printers. 

Today, most image capture devices have three color 
sensors which convert a multi-dimensional spectral sig- 
nal representing a scene into the output of three color 
channels: red (R), green (G), and blue (B). The RGB 
values of the image capture device are ultimately trans- 
formed into device representations of the color display 
(RGB) or the color printer (CMYK). The methods for 
achieving this transformation are a critical part of any 
color Image editing system. 

Prior art CI E systems are based on transformations 
of device-dependent RGB values into human-based 
representations, such as CIE XYZ. These intermediate 
XY2 values are then transformed into display RGB val- 
ues via a 3x3 transformation rriatrix or into printer CMYK 
values via a look-up table. This intermediate transfor- 
mation facilitates matches in color appearance between 
the original image or scene and the rendered image or 
scene. If the XYZ values in the original scene nnatch the 
XYZ values in the rendered scene, point by point, then 
the color appearance of the original and rendered scene 
will match. If the XYZ values between the original and 
rendered scene do not match on a point by point basis, 
there is no guarantee that the two scenes will appear 
identical. 

^^ These prior art CI E systems have several disadvan- 
tages. First, XYZ matching assumes that the illumlnant 
in the original and the rendered scene are identical. 
While it may be possible to design image capture devic- 
es that generate device RGB values that are within a 
linear transfonration of XYZ values for surfaces under 
one illuminant, it is not possible to design image capture 
devices that generate device RGB values that are within 
linear transformations of XYZ values for surfaces under 
\ any arbitrary or uncontrolled illuminant. Thus, the map- 
ping from the inr^ge capture device RGB values into the 
human -based XYZ values will always be accompanied 
by some information loss and, consequently, a color ap- 
pearance mismatch. 

Furthermore, prior art CIE systems do not separate 
the effects of the illuminant spectral power distribution 
(SPD) from the effects of the surface reflectances on the 
image capture sensors. As a result, they manipulate im- 
age properties derived from CIE XYZ, representations, 



such as hue, saturation, and brightness. These manip- 
ulations, however perceptual in nature, must be leamed 
by operators. 

A color Image editing system that allowed for color 
5 manipulation and adjustments based on illumination is 
desirable. It would be a further benefit if the manipula- 
tions and adjustments were intuitive to humans by cor- 
responding to change in lighting and surface color 

Accordingly, the invention proposes a method for 
10 characterizing a scene as surfaces and illuminants com- 
prising the steps of; 

measuring an illuminant spectral signature of a cap- 
turing device; 

fs capturing the scene as an array of pixel values; 

characterizing a scene transfer function by combin- 
ing the i jluminant spec tiaLjsigDaLure^wlth a device 
signature of the capturing device: and 
characterizing a scene signature by describing lllu- 

20 mination and surfaces corresponding to the array 
of pixel values according to the scene transfer func- 
tion. 

A spectral-based color image editing (SBCIE) sys- 
2S tem provides a general framework for image correction, 
composition, and enhancement. An operator can cor- 
rect images that were captured under bad illumination, 
combine images captured under different illumination, 
or select illuminants that make the image visually ap- 
30 pealing. Image manipulations are intuitive to humans 
because they correspond to changes in light and sur- 
face color. 

A color image editing system and method uses sen- 
sor responses of an Image capture device to estimate 

35 and separate lighting and surface reflectances in a 
scene. The system uses the estimated surface reflect- 
ances, along with a desired illuminant spectral power 
distribution (SPD), to calculate the corresponding XYZ 
values. To render the images on an imaging device, the 

40 XYZ values are converted into device-dependent dis- 
play or printer values. 

An image capture device, such as a scanner or dig- 
ital camera, represents a scene as image data that is 
resolved as device-dependent channels, such as RGB. 

45 For each channel, the spectral reflectances of the sur- 
faces within the scene are described as functions that 
are independent of a lighting function. The lighting func- 
tion describes illuminant spectral power distributran 
(SPD). 

50 First, an illuminant spectral signature of the Image 
capture device is measured usinc^ a spectrophotometric 
sensor or the lighting function is estimated. Next, a 
scene is captured as image data. A scene transfer func- 
tion is generated as a combination of the lighting func- 

55 tion and the device signature of the image capture de- 
vice. The image capture data is transfomned according 
to the scene transfer function to generate a scene sig- ' 
nature thai unquely describes the scene without the ef- 
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^ feet of the image capture device. Display operators can 
independently adjust tlio surface and ligliting compo- 
nents of the scene signature to achieve a desired image. 

The invention will now be described with reference 
to exemplary embodiments thereof shown in the accom- 
panying drawings, in which; 

Figure 1 illustrates an embodiment for a SBCIE sys- 
tem. 

Figure 2 illustrates a flowchart for the SBCIE system 
shown in Figure 1. 

Figure 3 illustrates a flowchart for step 50 shown in 
Figure 2. 

Figure 4 illustrates another embodiment for the SB- 
CIE system. 

Figure 5 illustrates a flowchart for the embodiment 
shown in Figure 4. 

Figure 6 illustrates an alternate embodiment for the 
SBCIE system. 

Figures 7A-B Illustrate a flowchart for embodiment 
shown in Figure 6. 

A spectral-based color image editing (SBCIE) sys- 
tem provides a general framework for image correction, 
composition, and enhancement. An operator can cor- 
rect images that were captured under bad illumination, 
combine images captured under different illumination, 
or select ilium inants that make the image visually ap- 
pealing. Image manipulations are intuitive to humans 
because they con-espond to changes in light and sur- 
face cobr. 

The spectral power distribution (SPD) of an illumi- 
nant can be described by a single vector, e, having en- 
tries which represent the amount of energy emitted over 
a range of wavelengths. For each surface, a spectral 
radiance factor, the light spectra that is emitted from the 
surface, has both diffuse and fluorescent components 
whk:h are described as a n by n matrix, S, where n is 
the number of entries representing the f unctbn values 
over a range of wavelengths. For example, n = 31 when 
the function values are sampled between 400nm and 
700 nm in 10 nm steps. The diffuse components have 
values solely along the diagonal vector s of the matrix 
S. Fluorescing components, which absorb light in one 
wavelength and emit light in a longer wavelength, can- 
not be described by a single vector 

The combined spectral signature of multiple illumi- 
nants, e, can be approximated by a linear combination 
of a smaller set of spectral basis functions, B,, where f 
is the number of basis functions and W| are the weights 
chosen to minimize the square error between the illumi- 
nant and its linear model approximation. 
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The spectral signatures of diffuse surfaces, s, may 



be described by a linear combination of a smaller set of 
spectral basis functbns. 



i = l 



These low-dimensional linear nnodels of illuminant spec- 
tra and surface reflectances are efficient and reduce the 
anrrount of stored data required to reconstruct the spec- 
tral representations. 

Similarly, the spectral signature of a digital camera 
in terms of RGB values tends to be low-dimensional 
spectral data. The dimensionality of the image capture 
data limits the dimensionality of the spectral reflectance 
functions. The accuracy of the spectral signatures of 
surfaces and illuminants is therefore limited by the in- 
herent dimensionality of the illuminant spectra and sur- 
face reflectances, the dimensionality of the image cap- 
ture data, and current knowledge about surfaces and 
illuminant spectra. ^ 

Figure 1 illustrates an embodiment for a SBCIE sys- / 
tem 10. A spectrophotometric sensor 12 is connected 
to an Image processor 14. The image processor 14 is 
further connected to a digital camera 16 and a display 
monitor 1 8. A printer 20 is connected to the image proc- 
essor 14 and the display monitor 18. 

In this embodiment, the spectrophotometric sensor 
12, such as that disclosed by Vincent in U.S. P.N. 
5,1 44,498, is based on a wedge filter that records both 
the color signal and the illuminant SPD such that it is 
possible to estimate the surface reflectances. The digital 
camera 16 is a DCS200 digital camera offered by the 
Kodak Company. The digital camera 1 6 has three sen- 
sors and a known device signature. 

Figure 2 illustrates a process flow chart for the SS^ 
CI E system 1 0 shown in Figure 1 . The method for gen- 
erating representatrons of spectral reflectances in 
scenes is based on knowledge of or estimation of the [ 
illuminant spectral power distribution, knowledge of the 1 
spatial distributbn and spectral sensitivities of f camera \ 
sensors, f camera sensor responses for surfaces in the I 
illuminated scene, and assumption about the f basis \ T^^ t^'^' 
function for surface spectral reflectances. ' 

In step 1 1 0, the illuminants in a scene are stored as 
an illuminant spectral signature, ^. A preferred method 
for gathering illuminant information is to directly meas- 
ure it using a spectrophotometric sensor. An altemate 
method is to restrict the dimensionality of the linear vec- 
tor space for representing the Illuminant SPD, such as 
Buchsbaum disclosed in "A spatial processor model for 
object color perception". Journal of the Franklin Insti- 
tute. Vol 310. pp. 1-26. 1980. and then to estimate the 
illuminant SPD from the distributbn of color pixels in the 
captured image. The perfornr^ance of the estimation al- 
gorithms depend upon the number of classes of color 



3 

05/20/2004, EAST Version: 1.4.1 



5 



EP 0 781 036 A1 



6 



sensors. The largerthe number of color sensor classes, 
the more accurate the illuminant SPD estimates are. 

In step 120, the illuminant spectral signature, e*. is 
transformed into a diagonal matrix, L, with nxn vectors. 
In step 130, the illuminant matrix, L, afong with two 
stored functions that describe the spectral sensitivities 
of the digital cameras, T, and 3 spectral surface basis 
functions, B, are multiplied to create a scene transfer 
matrix, C, that describgsall ima ges captui^ d^b y the dig- 
i tal camera , T, where illuminant and surface infonnation 
are decoupled. In this embodiment, L is a nxn matrix 
having values solely along the diagonal, T is a 3xn ma- 
trix, and B is a nx3 matrix. The resulting scene transfer 
matrix, C, is a 3x3 matrbc. In a general form that de- 
scribe%4 sensors, L is a nxn matrix, T is a 3xt matrix, B 
is a^fj^matrix, and C is a fxf matrix. ^^"^ 

In step 1 40, a matrix A is created by inverting the 
matrix C. In step 150, an image of the scene is captured 
and stored as a 3xp matrix of sensor pixel values, R. In 
step 1 60, a nxp scene matrix, S, t hat describes th e spec- 
tral reflectance of all points sampled , is constructed by 
multiplying 3 matrices: B, a nx3 matrix representing the 
3 spectral basis functions for surface reflectances, the 
inverted scene transfer matrix A, and the sensor pixel 
value matrix R. In this step, the system operator may 
change the spectral representations of surfaces in the 
scene, S, such as changes in the skin tone of a person, 
the color of a dress, the saturation of grass, or the color 
of drapes. 

In step 170, the spectral representation of scene il- 
lumination nnay be altered manually or automatically by 
selecting a jiew scene illum iriant vector, u'. The new 
scene illuminant vector, CT, ^transformed into an nxn 
diagonal matrix, L In step 180, L' is combined with S 
andLH)o calculate the XYZ values that should be dis- 
played or printed. When done manually, the illuminant 
SPD or surface reflectance can be changed until the im- 
age is visually appealing, such as in the color correction 
scheme proposed by Vrtiel and Trussell in "Color cor- 
rection using principal components", Cofor Research 
and Application, Vol. 1 7, pp. 329-337, 1 992, In step 1 90, 
the image is rendered. 

Figure 3 illustrates a flowchart for rendering the 
scene. In step 191, a printer spectral signature and a 
display spectral signature are determined. In step 193, 
the scene function, S, is customized to account for the 
display spectral signature. In step 1 95, the specific im- 
age may be rendered according to a look-up-table or 
specific function that corresponds to the printer spectral 
signature. 

Figure 4 illustrates an alternate embodiment for the 
SBCIE system 10'. The image processor 14 is connect- 
ed to the printer 20, the display monitor 18, and a color 
scanner 22. Scanner RGB values are used to estimate 
the weight factors of three reflectance basis functions. 
The estimated weights and the corresponding spectral 
basis functions are then used to build three-dimensional 
linear representations of surface spectral reflectances. 



This embodiment may be generalized to scanners with 
f spectral channels, estimating weights for f surface re- 
flectance basis functions. 

Figure 5 outlines a process flow chart for the em- 
s bodiment shown in Figure 4. 

In step 110, a color document is scanned, thereby 
creating a 3xp matrix of sensor responses, R. In step 
120, a surface representation is constructed for each of 
the pixels in the scanned document by multiplying the 
3xp matrix R with a stored function, Y The matrix mul- 
tiplication in step 120 can be stored in the scanner 
firmware as a nx3 matrix transformation applied to all 
scanner RGB values. 

One approach to matrix multiplication is as follows. 
T is a 3xn matrix that describes the spectral responsrv- 
ities of a three-channel scanner for a range of wave- 
length samples. T corresponds to the point-by-point 
product of the spectral power distribution of the scanner 
illuminant, the spectral transmittances of apparatus 
within the optical path, and the spectral responsrvity of 
the CCD scanner sensor. B is a nx3 matrix representing 
the 3 spectral basis functions for suriace reflectances. 
C is a matrix created by multiplying T with B. A is a 3x3 
matrix created by inverting C. Y is created by multiplying 
B and A. 

In step 1 30, the spectral representation of an illumi- 
nant, u', is selected and transformed into an nxn diago- 
nal matrix, L'. In step 140, L' is combined with S and H 
to calculate the XYZ values that should be displayed or 
printed. 

Although this embodiment is based on a 3-channel 
output, as the number of spectral channels increases, 
the estimates of the spectral reflectances will improve. 
The estimates can be optimized by judicious selection 
of the spectral basis functions. 

Figure 6 illustrates another alternate embodiment 
for the SBCIE system 10". The image processor 14 is 
connected between the printer 20 and the display mon- 
itor 18. Color images of scenes are often rendered on 
emissive or reflective displays. Even though image gen- 
eration is unknown, the operator has a perception that 
the displayed images are realistic depictions of actual 
scenes. One key area in computer graphics is devoted 
to developing realistic spectral representations of sur- 
faces and iiluminants to be rendered on the display One 
method is to have the operator select a region of the 
scene that corresponds to a white surface. Since a white 
surface has known spectral reflectance, an estimate of 
the illuminant spectral power distribution consistent with 
the displayed tristimulus values for the white surface can 
be made. 

Figures 7A and B illustrate a flowchart for the em- 
bodiment shown in Figure 6. In step 110, a pixel region 
that corresponds to a known surface, such as a white 
or neutral surface, is selected. The selection of this re- 
gion can be accomplished manually (by the user) or au- 
tomatically (by a programmed rule, such as computing 
the mean RGB values over the entire region, as in the 
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"gray-world" assumption described by Buchsbaum). 
The mean RGB values of these pixels are stored as a 
3x1 vector w. In step 120, the display gamma lookup 
table (LUT) is used to convert w to linear RGB values, 
w'. In step 130, w* is nnapped into the displayed tristim- 
utus values by multiplication with a stored 3x3 matrix 
generated by multiplying Z, a stored 3xn matrix contain- 
ing the CIE XYZ color matching functions, H, and a nx3 
matrix containing the spectral power distribution of the 
three display phosphors, M. 

In step 140, an llluminant, e', Is chosen that gener- 
ates the tristimulus values k when It Is reflected from the 
known surface. One way is provide a database of lllu- 
minant SPDs to find e' that minimizes the difference be- 
tween the predicted tristimulus values, k*, and the actual 
displayed tristimulus values, k. In step 1 40, an illumlnant 
e' is chosen from a database. In step 1 50, k* is computed 
by multiplying e' with a nxn function representing the 
spectral reflectances of the known surface, S^, and the 
stored function, H. In step 160, the predicted tristimulus 
values, k" and the actual displayed tristimulus values k 
are compared. If the difference between k and k' is less 
than a stored threshold, e' is passed as a vector to step 
170 where It is transformed into a nxn diagonal matrix, 
L'. 

In step 1 80, a scene transfer matrix, C, is generated 
by multiplying two stored functions: B, a nx3 matrix rep- 
resenting the surface spectral basis functions and H, a 
3xn matrix representing the CIE XYZ color matching 
functions with LV In step 90, the matrix C is inverted. 

In step 200, the displayed 3xp RGB values for all p 
pixels in the image are passed through the LUT to gen- 
erate RGB values that are linear with displayed lumi- 
nance. In step 210, the linear RGB values, R', are con- 
verted into XYZ values by multiplying R' with Z to gen- 
erate X. 

In step 220, a representation of spectral reflectanc- 
es of all points in the image is constructed. This is ac- 
complished by matrix multiplication of B, a stored nx3 
matrix, with A (the inverted scene transfer matrix), and 
X (a 3xp matrix containing the tristimulus values for alt 
surfaces depicted in the scene). 

In step 230, a new scene illuminant, u\ is chosen 
and transformed into a nxn diagonal matrix, L'. As in the 
previous embodiments, the system operator may 
change the spectral representations of surfaces in the 
scene, S, in step 220, or the spectral representation of 
scene illumination, u'. 

In step 240, L' is combined with S and H to calculate 
the XYZ values that should be displayed or printed. 
These XYZ values are then sent to appropriate LUTs for 
subsequent display or printing. In step 250, the image 
is rendered. 

Although the invention has been described in three 
channel RGB space. It would be obvious to one skilled 
in the art to Increase the number of the channels or 
translate the Image data Into a different visual represen- 
tation such as CMY 



Claims 

1. A method for characterizing a scene as surfaces 
and illuminants comprising the steps of: 

5 

measuring an llluminant spectral signature of a 
capturing device 110; 

capturing the scene as an array of pixel values 
150; 

10 characterizing a scene transfer function by 

combining the illuminant spectral signature with 
a device signature of the capturing device 130; 
and 

characterizing a scene signature by describing 
15 illumination and surfaces corresponding to the 

anay of pixel values according to the scene 
transfer function 1 60. 

2. A method for characterizing a scene as surfaces 
20 and illuminants as defined in claim 1 further com- 
prising the step of rendering the scene 1 90. 

3. A method for characterizing a scene as surfaces 
and Illuminants. as defined in claim 2. wherein the 

25 step of rendering further comprises the steps of: 

customizing the scene signature to describe 
the surfaces in the scene 160; 
customizing the scene signature according to 
30 an effective illuminant level 170; and 

rendering the scene signature 190. 

4. A method for characterizing a scene as surfaces 
and illuminants, as defined in claim 1, wherein the 

35 step of capturing the scene comprises the step of 
scanning a rendered image 110. 

5. A method for characterizing a scene as surfaces 
and illuminants, as defined in claim 3. wherein the 

^0 step of customizing the scene signature to describe 
the surfaces in the scene further comprises the step 
of characterizing the scene signature according to 
a look-up table 1 95. 

45 6. A method for characterizing a scene as surfaces 
and illuminants, as defined in claim 3, v\^erein the 
step of customizing the scene signature to describe 
the surfaces in the scene further comprises the step 
of rendering the scene surface matrix according to 
50 a transfer function that corresponds to a display de- 
vice signature 1 93. 

7. A method for characterizing a scene as surfaces 
and illuminants, as defined in claim 1 , further com- 

55 prising the step of capturing the device signature 
130. 

8. A method for characterizing a scene as surfaces 
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and illuminants, as defined in claim 1, further com- 
prising the step of modeling a set of surfaces as a 
surface spectral signature 180. 

A method for characterizing a scene as surfaces 
and illuminants, as defined in claim 8, wherein the 
step of modeling a set of surfaces further comprises 
the steps of: 

scanning a rendered scene 110; 

generating an array of pixels that corresponds 

to the rendered scene 200; and 

determining a scene signature according to the 

array of pixels and the device signature 230. 
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